Analysis
Cleaning Data
- .mat to .csv file for fasten analysis process
- select elemDataI and save into .mat format
- select necessary variables for analysis
- filter disposition file for artists/menu task
- check if simulation contains three instances
- if both non-restart and restart file contains 3 instances, used non-restart file (Participant 7 YP, Participant 21 ZM, Participant 123 ZM)
Artist/ Menu Task Detection
Find Engagement point
We used AMOC method in changepoint library with assumption that there is single potential changepoint from time when task was available to participant to when task was correctly completed.
We determined changepoint with mean and variance of speed and lane deviation. Then, selected minimum timeframe (faster changepoint detection result) as predicted engagement point for artist task.
- no predicted engagement point was after the participant was incorrect
Example 
exception Two simulations had one artist/menu task engaged (score didn’t change when task was available): Participant 123 YP (20140111102408.csv) and Participant 129 XM (20140104090136.csv) 
Define Control vs Experiment Group
Start of experiment group: using task availabe time: 
using predictedengagement time: 
Visuals
We found that total time (from task available to complete) is longer as participant experience more task. 
## # A tibble: 3 x 6
## eventNum min Q1 median Q3 max
## <fct> <int> <dbl> <dbl> <dbl> <int>
## 1 1 86 180. 282. 602. 619
## 2 2 161 366. 512. 603 612
## 3 3 245 450. 603 604 617Standard deviation of lane deviation by dosage group and secondary task engagement:
Conclusions:
- Not much difference when deriving engagement segment using changepoint detection vs. using entire period task is available
- We can still analyze the data both ways, but will use the changepoint detection approach for now
Boxplot version:
Conclusions:
- Standard Deviation of Lane Deviation is much higher when engaged in secondary task (artist-menu search)
- Similar differences across experimental conditions
- This visual ignores pairing of segments/participants
Paired (trt/control) version:
Conclusions:
- Within each experiment condition individual subject performance is worse when engaged in secondary task
Within Subject Performance Differences by Condition
Conclusions:
- Some (but not all) of the instances where performance was better while engaged in the secondary task had slower speeds during the task
- Difficult to gauge condition effects from this plot
Tables
Artist/Menu task segments available for analysis for each of the 19 subjects:
| XM | XP | YM | YP | ZM | ZP | Sum | |
|---|---|---|---|---|---|---|---|
| Subject ID 3 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 7 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 10 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 15 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 17 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 18 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 21 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 25 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 26 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 29 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 31 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 32 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 34 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 35 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 104 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 113 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 120 | 3 | 3 | 3 | 3 | 3 | 3 | 18 |
| Subject ID 123 | 3 | 3 | 3 | 2 | 3 | 3 | 17 |
| Subject ID 129 | 2 | 3 | 3 | 3 | 3 | 3 | 17 |
| Sum | 56 | 57 | 57 | 56 | 57 | 57 | 340 |
Conclusions:
- 19 subjects, only 2 missing events
Marginal Statistics
Marginally, there is a statistically significant effect of secondary task engagement (for most Dose/Event combos) on standard deviation of lane deviation (p-values are from paired t-tests within subject/event):
| DosingLevel | EventID | mean_SDL | sd_SDL | n | p_value | signif |
|---|---|---|---|---|---|---|
| XM | 1 | -0.1079823 | 0.6653311 | 19 | 0.47929 | |
| XM | 2 | 0.3852318 | 1.0004674 | 18 | 0.10234 | |
| XM | 3 | 0.6290988 | 0.9803059 | 19 | 0.00515 | ** |
| XP | 1 | 0.3818299 | 0.8399112 | 19 | 0.04753 | * |
| XP | 2 | 0.1759859 | 1.4630276 | 19 | 0.60005 | |
| XP | 3 | 0.4446652 | 0.8482441 | 19 | 0.02231 | * |
| YM | 1 | 0.2406506 | 0.9370421 | 19 | 0.26295 | |
| YM | 2 | -0.1298222 | 1.3342822 | 19 | 0.67149 | |
| YM | 3 | 0.4220633 | 0.7426300 | 19 | 0.01324 | * |
| YP | 1 | 0.4575804 | 0.8007496 | 19 | 0.01274 | * |
| YP | 2 | 0.5653850 | 1.1621832 | 18 | 0.03902 | * |
| YP | 3 | 0.3885585 | 0.7604960 | 19 | 0.02594 | * |
| ZM | 1 | 0.2998543 | 1.0046808 | 19 | 0.19328 | |
| ZM | 2 | 0.0252731 | 1.0177404 | 19 | 0.91380 | |
| ZM | 3 | 0.6257512 | 0.8121137 | 19 | 0.00078 | *** |
| ZP | 1 | 0.4451658 | 1.0555926 | 19 | 0.06603 | . |
| ZP | 2 | 0.5386620 | 1.0042601 | 19 | 0.01939 | * |
| ZP | 3 | 0.4225396 | 0.6069259 | 19 | 0.00241 | ** |
Models
Model 1 - Base Model
\[y_{ij} = \beta_0 + \gamma_i + \beta_1 SecTask_{ij} + \beta_2 DoseXM_{ij} + \beta_3 DoseYM_{ij} + \ldots + \epsilon_{ij}\]
- \(y_{ij}\) = Standard Deviation of lane deviation for subject \(i\) at dosing level \(j\)
- \(\gamma_i\) = subject specific random intercept
- \(SecTask_{ij}\) = dummy variable indicating observation measures performance during secondary task (1 = sec task, 0 = control segment)
- \(DoseAA_{ij}\) = dummy variable for dose = “AA” (1 = specified dose, 0 = reference group)
- In our models we will re-define the reference group such that all dose effects are positive
d3cp$Dose_Grp <- relevel(d3cp$DosingLevel, ref = "ZP")
fit <- lmer(data = d3cp, SD.Lane.Deviation ~ Dose_Grp + Experiment + (1 | ID))
fit_sum <- summary(fit)
kable(round(fit_sum$coefficients, 4))| Estimate | Std. Error | df | t value | Pr(>|t|) | |
|---|---|---|---|---|---|
| (Intercept) | 0.4598 | 0.0853 | 171.3512 | 5.3935 | 0.0000 |
| Dose_GrpXM | 0.0154 | 0.1012 | 655.1849 | 0.1522 | 0.8790 |
| Dose_GrpXP | 0.0756 | 0.1007 | 654.9280 | 0.7502 | 0.4534 |
| Dose_GrpYM | 0.0873 | 0.1007 | 654.9280 | 0.8665 | 0.3865 |
| Dose_GrpYP | 0.0517 | 0.1012 | 655.1849 | 0.5109 | 0.6096 |
| Dose_GrpZM | 0.0454 | 0.1007 | 654.9280 | 0.4507 | 0.6524 |
| Experiment | 0.3443 | 0.0583 | 654.9280 | 5.9029 | 0.0000 |
kable(round(anova(fit), 4))| Sum Sq | Mean Sq | NumDF | DenDF | F value | Pr(>F) | |
|---|---|---|---|---|---|---|
| Dose_Grp | 0.6434 | 0.1287 | 5 | 655.0996 | 0.2226 | 0.9528 |
| Experiment | 20.1476 | 20.1476 | 1 | 654.9280 | 34.8437 | 0.0000 |
Conclusions:
- Significant effect of secondary task engagement on standard deviation of lane deviation
No effect of experimental condition
- A few problems with this model:
- Doesn’t adjust for differences in speed
- Skewed distribution of
SD.Lane.Deviation - Treats all 3 secondary tasks as the same
- It also isn’t looking for effect modification by
Dose_Grp
Model 2 - Log-transformed outcome, control for eventNum and Avg.Speed
fit <- lmer(data = d3cp, log2(SD.Lane.Deviation) ~ relevel(DosingLevel, ref = "XM") +
factor(eventNum) + Avg.Speed + Experiment + (1 | ID))
kable(round(summary(fit)$coefficients, 4))| Estimate | Std. Error | df | t value | Pr(>|t|) | |
|---|---|---|---|---|---|
| (Intercept) | -3.9636 | 0.5686 | 370.8770 | -6.9712 | 0.0000 |
| relevel(DosingLevel, ref = “XM”)XP | 0.1522 | 0.1954 | 652.9692 | 0.7789 | 0.4363 |
| relevel(DosingLevel, ref = “XM”)YM | 0.2630 | 0.1976 | 657.7072 | 1.3311 | 0.1836 |
| relevel(DosingLevel, ref = “XM”)YP | 0.1300 | 0.1964 | 653.7328 | 0.6617 | 0.5084 |
| relevel(DosingLevel, ref = “XM”)ZM | 0.3465 | 0.1972 | 657.0216 | 1.7573 | 0.0793 |
| relevel(DosingLevel, ref = “XM”)ZP | 0.0550 | 0.1969 | 656.3927 | 0.2792 | 0.7802 |
| factor(eventNum)2 | 0.4422 | 0.1426 | 664.1713 | 3.1003 | 0.0020 |
| factor(eventNum)3 | 0.3166 | 0.1417 | 662.7813 | 2.2341 | 0.0258 |
| Avg.Speed | 0.0288 | 0.0087 | 402.0684 | 3.3167 | 0.0010 |
| Experiment | 0.7464 | 0.1124 | 652.2426 | 6.6397 | 0.0000 |
kable(round(anova(fit), 4))| Sum Sq | Mean Sq | NumDF | DenDF | F value | Pr(>F) | |
|---|---|---|---|---|---|---|
| relevel(DosingLevel, ref = “XM”) | 9.1783 | 1.8357 | 5 | 655.6375 | 0.8545 | 0.5115 |
| factor(eventNum) | 21.6956 | 10.8478 | 2 | 659.0886 | 5.0497 | 0.0067 |
| Avg.Speed | 23.6309 | 23.6309 | 1 | 402.0684 | 11.0003 | 0.0010 |
| Experiment | 94.7050 | 94.7050 | 1 | 652.2426 | 44.0854 | 0.0000 |
Conclusions:
- Secondary task engagement, speed, and eventID are all important predictors of standard deviation of lane deviation
- Experimental condition is not significant when controlling for these other predictors
Model 3 - Log-transformed outcome, control for eventNum and Avg.Speed, test for effect modification
- Does
Dose_Grpmodify the performance deteroration seen during secondary tasks?
fit <- lmer(data = d3cp, log2(SD.Lane.Deviation) ~ relevel(DosingLevel, ref = "ZP") +
factor(eventNum) + Avg.Speed + Experiment +
relevel(DosingLevel, ref = "ZP"):Experiment + (1 | ID))
kable(round(summary(fit)$coefficients, 4))| Estimate | Std. Error | df | t value | Pr(>|t|) | |
|---|---|---|---|---|---|
| (Intercept) | -3.9386 | 0.5590 | 387.3314 | -7.0460 | 0.0000 |
| relevel(DosingLevel, ref = “ZP”)XM | 0.0083 | 0.2779 | 649.4002 | 0.0297 | 0.9763 |
| relevel(DosingLevel, ref = “ZP”)XP | 0.0782 | 0.2778 | 650.6645 | 0.2814 | 0.7785 |
| relevel(DosingLevel, ref = “ZP”)YM | 0.2091 | 0.2754 | 647.3339 | 0.7593 | 0.4479 |
| relevel(DosingLevel, ref = “ZP”)YP | 0.1210 | 0.2795 | 651.8014 | 0.4329 | 0.6652 |
| relevel(DosingLevel, ref = “ZP”)ZM | 0.4550 | 0.2754 | 647.2304 | 1.6523 | 0.0990 |
| factor(eventNum)2 | 0.4430 | 0.1430 | 659.1465 | 3.0971 | 0.0020 |
| factor(eventNum)3 | 0.3175 | 0.1421 | 657.7581 | 2.2334 | 0.0259 |
| Avg.Speed | 0.0286 | 0.0087 | 398.7192 | 3.2839 | 0.0011 |
| Experiment | 0.8304 | 0.2754 | 647.2321 | 3.0158 | 0.0027 |
| relevel(DosingLevel, ref = “ZP”)XM:Experiment | -0.1252 | 0.3911 | 647.2303 | -0.3201 | 0.7490 |
| relevel(DosingLevel, ref = “ZP”)XP:Experiment | 0.0399 | 0.3894 | 647.2418 | 0.1025 | 0.9184 |
| relevel(DosingLevel, ref = “ZP”)YM:Experiment | -0.0024 | 0.3894 | 647.2402 | -0.0063 | 0.9950 |
| relevel(DosingLevel, ref = “ZP”)YP:Experiment | -0.0902 | 0.3911 | 647.2304 | -0.2306 | 0.8177 |
| relevel(DosingLevel, ref = “ZP”)ZM:Experiment | -0.3269 | 0.3894 | 647.2665 | -0.8394 | 0.4015 |
kable(round(anova(fit),4))| Sum Sq | Mean Sq | NumDF | DenDF | F value | Pr(>F) | |
|---|---|---|---|---|---|---|
| relevel(DosingLevel, ref = “ZP”) | 8.1640 | 1.6328 | 5 | 649.3536 | 0.7556 | 0.5821 |
| factor(eventNum) | 21.7850 | 10.8925 | 2 | 654.0749 | 5.0409 | 0.0067 |
| Avg.Speed | 23.3021 | 23.3021 | 1 | 398.7192 | 10.7838 | 0.0011 |
| Experiment | 94.6682 | 94.6682 | 1 | 647.2411 | 43.8109 | 0.0000 |
| relevel(DosingLevel, ref = “ZP”):Experiment | 2.5570 | 0.5114 | 5 | 647.2534 | 0.2367 | 0.9463 |
Conclusions:
Dose_Grpdoesn’t appear to modify the effect of secondary task engagement
Model 4 - Paired Difference as Outcome
fit <- lmer(data = df, diff_SD_lane_pos ~ relevel(DosingLevel, ref = "YM") + factor(EventID) + diff_speed + (1 | SubjectID))
kable(round(summary(fit)$coefficients, 4))| Estimate | Std. Error | df | t value | Pr(>|t|) | |
|---|---|---|---|---|---|
| (Intercept) | 0.1210 | 0.1562 | 211.2718 | 0.7743 | 0.4396 |
| relevel(DosingLevel, ref = “YM”)XM | 0.1231 | 0.1778 | 312.8555 | 0.6922 | 0.4893 |
| relevel(DosingLevel, ref = “YM”)XP | 0.1565 | 0.1769 | 312.6196 | 0.8849 | 0.3769 |
| relevel(DosingLevel, ref = “YM”)YP | 0.2848 | 0.1778 | 312.7221 | 1.6015 | 0.1103 |
| relevel(DosingLevel, ref = “YM”)ZM | 0.1390 | 0.1776 | 312.9546 | 0.7822 | 0.4347 |
| relevel(DosingLevel, ref = “YM”)ZP | 0.2910 | 0.1770 | 312.6593 | 1.6443 | 0.1011 |
| factor(EventID)2 | -0.0315 | 0.1305 | 315.2556 | -0.2413 | 0.8095 |
| factor(EventID)3 | 0.2020 | 0.1279 | 314.3777 | 1.5790 | 0.1153 |
| diff_speed | -0.0004 | 0.0178 | 330.3307 | -0.0235 | 0.9813 |
kable(round(anova(fit),4))| Sum Sq | Mean Sq | NumDF | DenDF | F value | Pr(>F) | |
|---|---|---|---|---|---|---|
| relevel(DosingLevel, ref = “YM”) | 3.3992 | 0.6798 | 5 | 312.8148 | 0.7622 | 0.5776 |
| factor(EventID) | 3.6341 | 1.8170 | 2 | 314.1180 | 2.0372 | 0.1321 |
| diff_speed | 0.0005 | 0.0005 | 1 | 330.3307 | 0.0006 | 0.9813 |
Conclusions:
- Little substantive difference analyzing the data this way
Model 5 - Avg Speed as Outcome
fit <- lmer(data = d3cp, Avg.Speed ~ relevel(DosingLevel, ref = "YP") +
factor(eventNum) + Experiment + (1 | ID))
kable(round(summary(fit)$coefficients, 4))| Estimate | Std. Error | df | t value | Pr(>|t|) | |
|---|---|---|---|---|---|
| (Intercept) | 63.3018 | 1.2485 | 34.3184 | 50.7011 | 0.0000 |
| relevel(DosingLevel, ref = “YP”)XM | -1.5575 | 0.8088 | 653.0613 | -1.9257 | 0.0546 |
| relevel(DosingLevel, ref = “YP”)XP | -0.2623 | 0.8051 | 653.0257 | -0.3258 | 0.7447 |
| relevel(DosingLevel, ref = “YP”)YM | -5.2056 | 0.8051 | 653.0257 | -6.4659 | 0.0000 |
| relevel(DosingLevel, ref = “YP”)ZM | -4.9229 | 0.8051 | 653.0257 | -6.1148 | 0.0000 |
| relevel(DosingLevel, ref = “YP”)ZP | -4.6509 | 0.8051 | 653.0257 | -5.7769 | 0.0000 |
| factor(eventNum)2 | 4.2126 | 0.5693 | 653.0241 | 7.3998 | 0.0000 |
| factor(eventNum)3 | 4.0525 | 0.5666 | 652.9927 | 7.1522 | 0.0000 |
| Experiment | 0.0939 | 0.4640 | 652.9927 | 0.2025 | 0.8396 |
kable(round(anova(fit),4))| Sum Sq | Mean Sq | NumDF | DenDF | F value | Pr(>F) | |
|---|---|---|---|---|---|---|
| relevel(DosingLevel, ref = “YP”) | 3347.6698 | 669.5340 | 5 | 653.0148 | 18.2942 | 0.0000 |
| factor(eventNum) | 2589.6467 | 1294.8233 | 2 | 653.0136 | 35.3794 | 0.0000 |
| Experiment | 1.5003 | 1.5003 | 1 | 652.9927 | 0.0410 | 0.8396 |
Conclusions:
- Dosing condition has a highly significant impact on participant speed
- Secondary task engagement doesn’t predict speed
- No interaction between dosing and secondary task engagment (model testing this not shown, p > .9)